Object Detection Using Radar And Vision Defined Image Detection Zone

ABSTRACT

An object-detection system includes a radar sensor, a camera, and a controller. The radar-sensor is suitable for mounting on a vehicle and is used to detect a radar-signal reflected by an object in a radar-field-of-view. The radar-signal is indicative of a range, range-rate, and a direction to the object relative to the vehicle. The camera is used to capture an image of a camera-field-of-view that overlaps the radar-field-of-view. The controller is in communication with the radar-sensor and the camera. The controller is configured to determine a range-map for the image based on the range and the direction of the radar detection, define a detection-zone in the image based on the range-map, and process only the detection-zone of the image to determine an identity of the object.

TECHNICAL FIELD OF INVENTION

This disclosure generally relates to an object-detection system, andmore particularly relates to information from a radar sensor to define adetection-zone within an image from a camera that is processed todetermine an identity of an object.

BACKGROUND OF INVENTION

Current Advanced Driver Assistance Systems (ADAS) and automated drivingsystem use multiple sensors, such as camera, radar, and LiDAR, to detectobjects in the proximity of the vehicle. The sensors use features suchas intensity, range, color etc. to detect objects. Range valuedescribing the distance to point in the scene is critical for thesuccess and reliability of object detection. Range values can becombined to generate a 2-D range-map showing the distance to points in ascene from a specific point. Range-map normally associated with a sensordevice such as camera or LiDAR. If the sensor is probably calibrated,range values can be given directly in feet or meters.

A single camera can be used to create generate range information usingstructure from motion. This typically results in sparse range estimationthat may not be accurate especially for dynamic objects. Stereo cameraor multiple camera setup system can also be used but add cost to thesystem in both hardware, since it uses multiple cameras and need to beprobably calibrated, and in software since stereo range estimation is anexpensive and error prone process. LiDAR has been the most widely usedsensor for range estimation but is currently expensive for mostapplications, and limited in range it can measure.

SUMMARY OF THE INVENTION

Various embodiments describe herein include methods and systems forusing Radar or LiDAR information to improve vision algorithms, wheredetections, range information, and range-rate are used as input tovision algorithms.

In accordance with one embodiment, an object-detection system configuredto detect an object proximate to a vehicle is provided. The systemincludes a radar sensor, a camera, and a controller. The radar-sensor issuitable for mounting on a vehicle and is used to detect a radar-signalreflected by an object in a radar-field-of-view. The radar-signal isindicative of a range, range-rate, and a direction to the objectrelative to the vehicle. The camera is used to capture an image of acamera-field-of-view that overlaps the radar-field-of-view. Thecontroller is in communication with the radar-sensor and the camera. Thecontroller is configured to determine a range-map for the image based onthe RDU range and the direction, define a detection-zone in the imagebased on the range-map, and process only the detection-zone of the imageto determine an identity of the object.

In one exemplary embodiment of the present invention, a range-map isdetermined from RDU range information. The range-map can be used byvision algorithms to decide on the scale of search to use, ondetermining time-to-contact (TTC), for properly placing visiondetection-zones on the ground among others.

In an alternative embodiment of the present invention, the speed of theobject in the image is determined from RDU range-rate information. Thishelp vision tracking by limiting the search space in the next frame. Itcan also be used to improve classification results of the visionalgorithms. For example, a high speed object cannot be classified as apedestrian.

In yet another embodiment of the present invention, an occupancy-gridand an interval-map are generated from the range-map. Vision systemcannot provide, in general, good estimation of the range-map and hencethe use of RDU range can help generate accurate occupancy-grid orinterval-maps. In addition to generating static occupancy andinterval-maps, RDU range-rate can be used to provide dynamicinformation.

Further features and advantages will appear more clearly on a reading ofthe following detailed description of the preferred embodiment, which isgiven by way of non-limiting example only and with reference to theaccompanying drawings.

BRIEF DESCRIPTION OF DRAWINGS

The present invention will now be described, by way of example withreference to the accompanying drawings, in which:

FIG. 1 depicts a block diagram of a system of for combining radar andvision sensors;

FIG. 2 depicts an object detection flow diagram of the system of FIG. 1;

FIG. 3 depict an example of constructing a region of interest based onRadar detections by the system of FIG. 1;

FIGS. 4A and 4B depict two examples of multi-scale object detection bythe system of FIG. 1;

FIGS. 5A and 5B depict two examples of building range-map from RDU rangeinformation by the system of FIG. 1;

FIG. 6 depicts an example of constructing a range-map using RDU rangeinformation by the system of FIG. 1;

FIGS. 7A and 7B depict an illustration of tracking using RDU range-rateby the system of FIG. 1; and

FIGS. 8A and 8B depicts examples of occupancy-grid and interval-map bythe system of FIG. 1.

It should be understood that the drawings are for purposes ofillustrating the concepts of the invention and are not necessarily theonly possible configuration for illustrating the invention.

DETAILED DESCRIPTION

The present principles advantageously provide a method and system forimproving vision detection, classification and tracking based on RangeDetecting Unit (RDU). Although the present principles will be describedprimarily within the context of using Radar, the specific embodiments ofthe present invention should not be treated as limiting in the scope ofthe invention. For example, in an alternative embodiment of the presentinvention, a LiDAR unit may be used instead of Radar unit or a LightEmitting Diode (LED) based sensor may be used.

In ADAS and automated driving systems, sensors are used to detect,classify, and track obstacles around the host vehicle. Objects can beVehicles, Pedestrian, or unknown class referred to as general objects.Typically two or more sensors are used to overcome the shortcoming ofsingle sensor and to increase the reliability of object detection,classifications, and tracking. The outputs of the sensors are thencombined to determine the list of objects in the scene. Combining sensorinformation can be done at a high level where every sensor is processedindependently with results combined at the end or at a low level whereone sensor is used by another sensor at an early stage of processing. Acombination of these methods is also possible. Without loss ofgenerality, the system and method presented herein focuses on ADAS andRadar. In general, LiDAR provide more accurate and denser data and hencecan result in better performance than Radar.

FIG. 1 illustrates a non-limiting example of an object-detection system900 configured to detect an object 902 proximate to a vehicle 924. Theobject-detection system 900 includes a radar-sensor 904 that is used todetect a radar-signal 926 reflected by an object in aradar-field-of-view 906. The object-detection system 900 also includes acamera 908 used to capture an image 302 (FIG. 3A) of the object 902 in acamera-field-of-view 910 that overlaps the radar-field-of-view 906. Theobject-detection system 900 is generally configured to combineinformation from the radar-sensor 904 and the camera 908 in a mannerthat takes advantage of the strengths of these two devices and therebycompensating for the weaknesses of the radar-sensor 904 and the camera908. One advantage of a camera based system is accurate detection andclassification of an object. However, it is difficult for a camera onlybased system to estimate accurately the range and speed of the object.Radar or LiDAR can provide accurate range and range-rate and hence canenhance object detection system.

The controller 912 is further configured to use the RDU range 928 (therange 928), RDU direction (the direction 930), and an RDU range-rate 932(the range-rate 932) to build a range-map 600 and use the range-map toadjust the size and location 606 of the detection-zone 602, to adjustthe scale-factor 400 used for object search, to compute the TTC 608 tothe detection-zone 602, to compute a static or dynamic instance of anoccupancy-grid 800, and compute a static or dynamic instance of aninterval-map 802. The controller 912 is also configured to use the RDUrange-rate 932 to estimate the temporal movement path of the object forthe purpose of reducing the search space during object tracking.

FIG. 2 shows a non-limiting example of a signal flow diagram forcombining Radar and vision. In the figure, Radar provides information202 to the vision system 204 early in algorithm processing to allow itto detect, classify, and track objects effectively. The most usefulinformation that Radar can provide in addition to RDUs are the RDU rangeand RDU range-rate. This information can benefit vision algorithms in anumber of ways as will be described below.

Monocular vision system has been popular for ADAS applications due toits low cost and effectiveness in determining object class. To detectobjects, a dataset of labeled image windows of fixed size (e.g. −n×m)are built. The database includes both positive and negative examples ofthe object to be detected (e.g. vehicle). A classifier is then trainedto tell these windows apart. Every n×m window is then passed to theclassifier for processing. Windows that the classifier labels positivecontain the object, and those labeled negative do not. Object detectionand classification can be done separately i.e. first detect thenclassify or detect/classify at the same. Detected objects are thentracked in the temporal domain using, for example, the centroid of theobject window.

The search for objects in vision systems is typically done in slidingwindow fashion [Computer vision a Modern approach, by Forsyth and Ponce,Publisher: Pearson, 2^(nd) edition 2011] starting from the top leftcorner of the image to the lower right. Since not all instances of anobject will be the same size in the image, search must be done overmultiple scales. Typically three or more scales are used depending onprocessing power available. We will use scale-factor 400 to representthe number of scales used in the search. The sliding window detection iswell behaved in practice with different applications requiring differentchoices of feature. However the processing requirements are very highwhich limits the types and number of objects that can be searched.

To speed up the search for objects, in one embodiment, RDUs are used toconstruct a boundary curve that divides the image into free and occupiedspace. The resulting occupied space is the only region that needs to besearched by the vision algorithm and since it is usually smaller thanthe full image; it results in significant processing time saving to thesliding window algorithm. FIG. 3 presents an example of free spacecreation from RDUs 300 where RDUs overlaid on the image 302 arerepresented as white squares. The resulting free space boundary curve isshown in FIG. 3B (304) overlaid on the image 302.

Range-map (or depth map) is important for many ADAS and autonomousvehicle applications. In computer vision, structure from motion orstereo camera can be used to estimate a range-map. This typically doesnot provide sufficient density in the case of structure from motion orexpensive using stereo camera in either hardware (two cameras) orsoftware (disparity algorithms). In this disclosure, RDU range is usedto build an approximate range-map. The range-map can be used to reducethe search space in the sliding window algorithm, to estimate Time tocontact (TTC), to probably place or resize vision detection-zone, or tobuild an occupancy-grid among others.

In one embodiment, the range-map is constructed as shown in FIG. 5A. Inthe figure, RDU detections are shown as a black square and gray scalevalues represent distance from the bottom of the image. The range-map isconstructed as follows. Let us use the range value at the lowest box 500of the image as zero. Considering the RDU 502 in the lower left box ofthe image, the range value of the RDU is used as the ceiling of therange value from the lowest box 500 of the image and the range isgradually increased from zero to the celling in the lowest box 500. Nowthe top location of the lowest box 500 in the image is used as the rangefloor value. The second square 504, is defined from the ceiling of thelowest box 500 to the RDU range value of the RDU detection in the middleright of the image. This process is repeated until the last RDU isreached. No range values are assigned if no RDUs are available such asthe top box 508.

In another embodiment, the range-map is defined as shown in FIG. 5B. Inthe figure, a range value is defined from RDU range value 512 andassigned to all pixels in the neighborhood of the detection such aspixels 514. The size of the neighborhood depends on whether there is adetected vision object in the neighborhood of the RDU detection or not,the type of radar, the distance from the camera among others. In thefigure, the detection can be slightly rotated to match the radar contourbehavior. The two methods can be combined as shown in FIG. 6 where themethod in FIG. 5A is used for short range and the method in FIG. 5B isused for far range.

The range-map has a number of uses in vision algorithm. In oneembodiment, the range-map is used to decide on the scale-factor 400 touse in the vision processing algorithm. As an example, only one or twoscales are used in the sliding window algorithm. In another example, therange-map is used to calculate the detection-zone size as will bedescribed below. This can result in significant saving in processingtime. In FIG. 4, two examples of multi-scale processing are shown. InFIG. 4A, the detection-zone is fixed while the image size is reduced.Three images (scales) are shown in the figure. In FIG. 4B, analternative implementation is shown with the detection-zone scaled whilethe image size is fixed.

In another embodiment, the range-map and range-rate are used for TTCcalculation. TTC uses the free space in front of the vehicle to estimatethe time to reach an object in front of the vehicle. TTC can be definedby; TTC=Range/Velocity, where range is estimated from the range-map andVelocity is defined as the relative speed of the host vehicle minus thespeed of the target object. The target object speed can be computed fromthe RDU range-rate.

In one embodiment, the location and size of the detection-zone can beadjusted based on the range-map. For example, it is well known in theart [Robust range estimation with a monocular camera for vision basedforward collision warning system, K. Park and S. Hwang, Scientific worldjournal January 2014], that if the real width of a vehicle is known, thevehicle width in the image can be defined using the formula: Vehiclewidth in image=Camera focal length*(Real Vehicle width/Z), where Z isthe range in front of the camera. Using certain value for real vehiclewidth such as 1.4 m -2 m, the width of the image can be estimated. Theprojection of object into the image can be estimated as follows. A pointin the road at Z will project to the image at a height y, where y isgiven by y=focal length*(camera height/Z). Since Z is known from therange-map, the height of the object y can be estimated. The analysisassumes that the camera is mounted so that the optical axis is parallelto the road surface. Similar analysis can be done if the camera ispitching down.

In an embodiment, RDU range-rate is used to improve tracking speed andaccuracy. In vision tracking, an object detected at time “t” can betracked at time “t+1” by searching the image at time “t+1” to find thebest match to the image window at time “t”. If no information on theobject speed is available, the search area would be large and hence timeconsuming. If the search space is large, you may also end up matchingthe wrong image window in the second frame. Using RDU range-rate andpossibly direction from RDU would help us better estimate how much anobject moved and hence would help us focus the search in a smaller areaof the image.

FIG. 7 illustrates the concept proposed in the above embodiment. In thefigure, the detected object 700 at time “t” shown as a black box istracked at time “t+1” 704. The direction 702 shown in the figure can becomputed from RDU range-rate and direction and allow us to define asubsequent-zone 706 in the image at time “t+1”. The search space orsubsequent-zone 706 boundary is marked in black in the figure.

In yet another embodiment, the RDU range-rate can also be used toinfluence the classification results. As an example if an object windowhas an RDU with large RDU range-rate (fast object) it can be used toincrease the confidence of a vehicle class vs. truck or motorcycle vs.bicycle.

As mentioned earlier, LiDAR can be used in similar way to compute freespace and range-map. Since LiDAR provides more accurate and denser data,free space and range-map can be estimated with higher accuracy. Theapproach can be very useful for sparse LiDAR where only limitedmeasurements are available during the LIDAR scan.

Occupancy-grid has been a popular environmental representation for anumber of ADAS and automated driving applications. LiDAR has been themost widely used sensor for building occupancy-grids since it providesmore accurate and denser measurements of occupancy. In thisrepresentation, the sensor field of view is tessellated into cells ofcertain size, for example, 20 cm×20 cm. Each cell is then classifiedinto occupied, free, or unknown. The classification typically done usingsensor measurements with a probability measure associated with theclassification. The probability varies from 0-1 with low probabilityassociated with low classification confidence and high probabilityassociated with high confidence. In one embodiment, RDUs are used todefine the occupancy of the grid by using the free space estimationdescribed above. However due to the sparseness of radar detection, manycells may not be defined correctly. Vision algorithm can produce asestimated of the occupancy from color, texture, gradient or other visualmeasures. However, vision does not provide an accurate measurement ofrange. Therefore in one preferred embodiment, the occupancy-grid state(occupied, free, or unknown) is computed from combining radar and visionfree space estimation. The range information needed for theoccupancy-grid is computed from the range-map.

The occupancy-grid as described above provides a static representationof the environment. In one embodiment, dynamic information is added tothe occupancy-grid from the RDU range-rate. The velocity information canbe very important for many applications. The use of RDU range-rateprovides more accurate dynamic information compared to estimating motionfrom vision directly. However, radar does not provide dense estimationand hence many cells in the occupancy-grid will have undefined dynamicstate information. FIG. 8A shows an example of a dynamic instance of theoccupancy-grid 800.

The interval-map 802 is another general environmental representation. Inthis presentation, the space around the vehicle is discretized in thelongitudinal direction. In another embodiment, an interval-map isconstructed similar to the occupancy-grid. In FIG. 8B, an example of adynamic instance of an interval-map 802 is shown.

While this invention has been described in terms of the preferredembodiments thereof, it is not intended to be so limited, but ratheronly to the extent set forth in the claims that follow.

We claim:
 1. An object-detection system configured to detect an objectproximate to a vehicle, said system comprising: a radar-sensor suitablefor mounting on a vehicle and used to detect a radar-signal reflected byan object in a radar-field-of-view, said radar-signal indicative of arange and a direction to the object relative to the vehicle; a cameraused to capture an image of a camera-field-of-view that overlaps theradar-field-of-view; and a controller in communication with theradar-sensor and the camera, said controller configured to determine arange-map for the image based on the range and the direction, define adetection-zone in the image based on the range-map, and process only thedetection-zone of the image to determine an identity of the object. 2.The system in accordance with claim 1, wherein the controller is furtherconfigured to adjust a size and a location of the detection-zone basedon the range-map.
 3. The system in accordance with claim 1, wherein thecontroller is further configured to determine a scale-factor based onthe range-map, said scale-factor used to determine the identity of theobject.
 4. The system in accordance with claim 1, wherein theradar-signal is also indicative of an RDU range-rate of the object, andthe controller is further configured to determine a TTC based on the RDUrange and the RDU range-rate.
 5. The system in accordance with claim 1,wherein the radar-signal is also indicative of a RDU range-rate of theobject, and the controller is further configured to determine theidentity of the object as one of a nearby-object and an occluded-objectbased on the RDU range and the RDU range-rate.
 6. The system inaccordance with claim 1, wherein the radar-signal is also indicative ofa RDU range-rate and a direction of the object, and the controller isfurther configured to determine a subsequent-zone in a subsequent-imagebased on the detection-zone, the RDU range-rate, and the direction. 7.The system in accordance with claim 1, wherein the controller is furtherconfigured to determine an occupancy-grid of the image based on therange-map.
 8. The system in accordance with claim 1, wherein theradar-signal is also indicative of a RDU range-rate of the object, andthe controller is further configured to determine an occupancy-grid ofthe image based on the range-map and the RDU range-rate.
 9. The systemin accordance with claim 1, wherein the controller is further configuredto determine an interval-map of the image based on the range-map. 10.The system in accordance with claim 1, wherein the radar-signal is alsoindicative of a range-rate of the object, and the controller is furtherconfigured to determine an interval-map of the image based on therange-map and the RDU range-rate.